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Abstract 


The  Automated  Grading  System  (AGS)  was  developed  jointly  by  Defence  Research  and 
Development  Canada  (DRDC)  Toronto  and  the  Canadian  Forces  School  of  Military  Intelligence 
(CFSMI)  to  provide  students  at  the  school  a  tool  to  help  in  the  composition  of  accurate  and 
effective  Intelligence  Summaries  (INTSUMs).  The  AGS  is  a  web-browse  based  system  that 
provides  feedback  to  students  about  how  well  their  summary  matches  that  of  a  gold  standard 
summary  written  by  an  instructor.  The  AGS  allows  students  to  iteratively  correct  and  re-submit 
their  summaries  as  they  attempt  to  maximize  the  match  between  their  summary  and  the  gold 
standard.  In  this  report,  we  provide  both  the  instructor  and  student  user’s  manual  for  the  AGS. 
Importantly,  we  also  provide  the  results  of  a  small  validation  study  wherein  we  asked  participants 
to  summarize  news  stories  about  sea  piracy  near  Somalia.  Participants  used  feedback  from  the 
AGS  to  improve  their  summaries  until  they  were  satisfied  that  they  had  done  the  best  job  they 
could  do.  The  grades  given  to  the  first  and  final  summaries  by  the  AGS  were  then  compared  to 
the  grades  awarded  by  the  lead  instructor  at  CFSMI.  The  tool  and  the  instructor’s  assessments  of 
the  summaries  were  in  close  agreement.  The  results  confirm  that  the  AGS  can  be  used  as  an 
effective  teaching  tool  to  help  students  improve  their  summary-writing  skills. 


Resume 


Le  systeme  automatique  de  notation  (AGS)  a  ete  developpe  en  collaboration  avec  Recherche  et 
developpement  pour  la  defense  Canada  (RDDC),  a  Toronto,  et  l’Ecole  du  renseignement  militaire 
des  Forces  canadiennes  (ERMFC)  dans  le  but  d’aider  les  eleves  a  rediger  des  resumes 
renseignement  (INTSUM)  efficaces  et  precis.  II  s’agit  d’un  logiciel  Web  permettant  aux  eleves  de 
savoir  dans  quelle  mesure  leurs  resumes  correspondent  au  modele  ideal  compose  par  un 
instructeur.  L’AGS  leur  permet  de  modifier  iterativement  leurs  textes  et  de  les  faire  analyser  de 
nouveau  pour  maximiser  la  correspondance.  Ce  rapport  comprend  un  manuel  d’utilisateur  pour 
les  instructeurs  et  les  eleves.  II  presente  egalement  les  resultats  d’une  petite  etude  de  validation 
durant  laquelle  les  participants  ont  resume  des  reportages  sur  la  piraterie  en  mer  pres  de  la 
Somalie.  Ils  ont  utilise  la  retroaction  de  TAGS  pour  retravailler  leurs  compositions  jusqu’a  ce 
qu’ils  jugent  avoir  fait  de  leur  mieux.  Les  notes  attribuees  par  le  logiciel  pour  les  premiers  et 
demiers  resumes  ont  ete  comparees  a  celles  donnees  par  l’instructeur  principal  de  l’ERMFC,  et  le 
tout  concordait.  Cela  confirme  que  l’AGS  est  un  outil  pedagogique  efficace  pouvant  aider  les 
eleves  a  ameliorer  leur  capacite  a  rediger  des  resumes. 
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Executive  summary 


Users’  Manual  and  Validation  of  the  Automated  Grading  System 
(AGS):  Improving  the  Quality  of  Intelligence  Summaries  Using 
Feedback  from  an  Unsupervised  Model  of  Semantics 

Peter  Kwantes;  Ron  Wulf;  Benjamin  Stone;  DRDC  Toronto  TM  2012-0602012- 

060;  Defence  R&D  Canada  -  Toronto;  December  2012. 

Introduction  or  background:  Intelligence  analysts  must  be  able  to  write  clear,  concise  and 
accurate  intelligence  summaries.  As  with  most  skills,  effective  summarization  skills  must  be 
learned.  The  Automated  Grading  System  (AGS)  is  a  computer  program  that  assigns  a  grade  and 
provides  basic  feedback  to  intelligence  summaries  written  by  students  learning  the  skill.  Briefly, 
the  computer  program  compares  the  semantic  content  of  a  student’s  summary  of  source 
documents  to  a  “gold  standard”  summary.  If  the  summary  and  gold  standard  are  similar  enough, 
the  student  receives  a  passing  grade.  Otherwise,  the  student  “fails”  and  is  invited  to  attempt  to 
improve  his  or  her  summary  and  resubmit.  In  this  report,  we  validated  the  AGS  by  having 
undergraduates  at  Ohio  State  University  write  a  summary  of  news  articles  on  the  topic  of  marine 
piracy.  Students  used  the  AGS  to  improve  their  summaries,  and  submitted  their  final  version 
when  they  were  satisfied  with  its  quality.  The  students’  first- written  and  final  summaries  were 
then  given  to  the  lead  analyst/instructor  at  the  Canadian  Forces  School  of  Military  Intelligence 
(CFSMI)  for  grading.  The  validation  involved  the  instructor  grading  the  two  summaries  provided 
by  each  student. 

Results:  In  general,  the  grades  assigned  to  the  summaries  by  the  AGS  agreed  with  those  provided 
by  the  instructor.  Importantly,  however,  the  AGS  and  instructor  had  high  agreement  on  the 
amount  with  which  summaries  improved  with  practice.  Put  simply,  the  AGS  was  an  effective  tool 
for  improving  the  quality  of  participants’  summaries. 

Significance:  This  validation  study  demonstrated  that  the  AGS  could  be  used  by  training  and 
educational  organizations  within  the  CF  as  a  tool  for  helping  students  improve  certain  aspects  of 
their  writing  ability.  In  particular,  the  educational  context  for  the  intelligence  domain  can  benefit 
from  using  the  AGS  to  help  students  learn  effective  intelligence  summary  writing.  Perhaps 
equally  important,  the  AGS  may  preclude  the  need  to  train  effective  writers  in  class.  Rather, 
because  the  program  is  browser-based  and  installed  on  a  network,  students  can  practice  summary 
writing  in  their  own  time  outside  of  class. 

Future  plans:  The  AGS’  intended  user  (and  co-developer)  was  the  CFSMI.  Our  intention  is  to 
have  the  program  installed  at  the  school  so  that  students  can  start  using  it  immediately.  The 
usefulness  of  the  AGS  likely  reaches  beyond  just  the  intelligence  realm.  Anyone  required  to  learn 
effective  reporting  could  potentially  benefit  from  using  the  AGS.  We  see  other  potential  users  that 
include  the  Royal  Military  College  of  Canada  (RMCC),  Canadian  Defence  Academy  (CDA)  and 
the  Influence  Activities  Task  Force  (IATF) 
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Sommaire 


Validation  of  the  Automated  Grading  System  (AGS):  Improving 
the  Quality  of  Intelligence  Summaries  Using  Feedback  from  an 
Unsupervised  Model  of  Semantics 

Peter  Kwantes;  Ron  Wulf;  Benjamin  Stone  ;  DRDC  Toronto  TM  2012-0602012- 

060  ;  R  &  D  pour  la  defense  Canada  -  Toronto;  decembre  2012. 

Introduction  ou  contexte  :  Les  analystes  du  renseignement  doivent  etre  capable  de  rediger  des 
resumes  renseignement  clairs,  concis  et  precis.  Comme  pour  toute  competence,  il  faut  apprendre  a 
resumer  de  maniere  efficace.  Le  systeme  automatique  de  notation  (AGS)  est  un  logiciel  donnant 
une  note  et  une  retroaction  de  base  aux  textes  des  eleves  en  apprentissage.  En  bref,  le  programme 
compare  le  contenu  semantique  du  resume  d’un  eleve  a  celui  d’un  modele  «  ideal ».  Lorsque  le 
contenu  est  semblable,  l’eleve  obtient  une  note  de  passage.  Par  contre,  s’il  «  echoue  »,  il  peut 
retravailler  sa  composition  avant  de  la  faire  analyser  de  nouveau.  La  participation  d’etudiants  de 
premier  cycle  de  l’universite  de  l’Ohio  a  permis  de  valider  l’AGS.  Apres  avoir  resume  des 
articles  sur  la  piraterie  en  mer,  les  participants  ont  utilise  le  logiciel  afin  d’ameliorer  leurs  textes 
et  de  presenter  la  version  definitive  une  fois  qu'ils  sont  satisfaits  de  la  qualite.  Les  premieres  et 
demieres  versions  des  resumes  ont  ensuite  ete  remises  a  l’analyste/instructeur  principal  de  l’Ecole 
du  renseignement  militaire  des  Forces  canadiennes  (ERMFC)  pour  qu’il  les  note.  Dans  le  cadre 
du  processus  de  validation,  il  a  attribue  des  notes  aux  deux  resumes  de  tous  les  etudiants. 

Resultats :  Dans  l’ensemble,  les  notes  donnees  par  l’AGS  concordaient  avec  celles  de 
l’instructeur.  Il  faut  noter  que  le  niveau  de  concordance  etait  eleve  en  ce  qui  a  trait  au  degre 
d’amelioration  des  resumes  resultant  de  la  pratique.  En  termes  simples,  l’AGS  est  un  outil 
efficace  permettant  d’augmenter  la  qualite  des  compositions  des  participants. 

Importance  :  Cette  etude  de  validation  demontre  que  des  organisations  d’enseignement  et 
d’instruction  des  FC  pourraient  utiliser  l’AGS  comme  outil  pour  aider  les  eleves  a  ameliorer 
certains  aspects  de  leur  capacite  de  redaction.  Le  contexte  educatif  du  domaine  du  renseignement 
pourrait  tout  particulierement  profiter  du  logiciel  pour  enseigner  la  redaction  de  resumes 
renseignement  efficaces.  En  outre,  l’AGS  pourrait  ecarter  la  necessite  de  former  des  redacteurs  en 
classe.  Puisqu’il  s’agit  d’un  logiciel  Web  installe  sur  un  reseau,  les  eleves  peuvent  s’exercer  a 
rediger  des  resumes  en  dehors  de  la  classe. 

Perspectives  :  L’AGS  a  ete  con£u  pour  l’ERMFC  (qui  a  collabore  au  developpement).  L’objectif 
est  d’installer  le  logiciel  a  l’ecole  afin  que  les  eleves  puissent  l’utiliser  immediatement.  Il  est 
probable  que  l’utilite  de  l’AGS  aille  au-dela  du  domaine  du  renseignement.  Tous  les  eleves 
apprenant  a  rediger  des  rapports  efficaces  pourraient  profiter  de  cet  outil.  Le  College  militaire 
royal  du  Canada  (CMRC),  l’Academie  canadienne  de  la  Defense  et  la  force  operationnelle 
chargee  des  activites  d’influence  (FO  AI)  font  partie  des  utilisateurs  potentiels. 
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1  Introduction 


1.1  Summarization  as  an  Intelligence  Operator’s  role 

Part  of  the  job  of  an  intelligence  analyst,  is  to  synthesize  the  content  of  several  documents  or 
reports  into  a  single,  concise  document  that  gets  passed  up  to  the  chain  of  command  to  an 
intelligence  consumer.  Clearly,  it  is  very  important  for  the  document,  called  an  intelligence 
summary  (or  INTSUM)  to  be  accurate  and  contain  all  the  necessary  details.  The  ability  to 
produce  high  quality  summaries  however,  is  a  skill  requiring  practice,  and  as  yet,  the  Canadian 
Forces  has  no  time-effective  means  by  which  to  teach  the  skill  within  the  training  and  educational 
contexts.  In  response,  personnel  from  Defence  Research  and  Development  Canada  -  Toronto 
(DRDC  -  Toronto)  and  the  Canadian  Forces  School  of  Military  Intelligence  (CFSMI) 
collaborated  to  develop  a  software  tool  called  the  Automated  Grading  System  (AGS)  that  allows 
students  to  practice  and  improve  INTSUM  writing  without  the  need  for  instructor  involvement. 

The  AGS  is  a  browser-based  application  with  which  a  student  reads  a  set  of  documents,  writes  a 
summary  and  submits  to  it  the  system  for  grading.  The  AGS  provides  the  students  with  a  set  of 
grades,  each  of  which  indicates  the  extent  to  which  certain  key  pieces  of  information  are  missing 
from  their  summaries  .The  grade  assignment  process  is  fairly  simple,  and  described  in  greater 
detail  in  the  next  section  below.  Briefly  however,  a  computational  model  of  semantics  is  trained 
on  a  large  collection  of  documents  to  build  “meaning”  representations  for  the  words  it  has 
encountered  in  the  collection.  These  word  representations  are  used  as  a  basis  for  creating  a 
semantic  representation  for  a  “gold  standard”  summary  document  written  by  the  AGS 
administrator  or,  which  will  most  often  be  the  case,  an  instructor.  When  a  student  submits  a 
summary  to  the  AGS,  a  semantic  representation  of  it  is  created  and  compared  to  that  of  the  gold 
standard.  If  the  similarity  between  the  two  does  not  exceed  some  threshold  set  by  the 
administrator,  the  student  fails  and  is  encouraged  to  improve  the  summary  and  re-submit  it. 

1.2  Purpose  and  Organization  of  this  Report 


The  AGS  is  described  in  detail  in  the  following  section.  Following  the  description,  we  describe  a 
validation  study  in  which  we  tested  the  AGS’s  efficacy  in  improving  INTSUMs  using  university 
undergraduates  as  participants.  To  foreshadow  the  results  of  the  study,  the  AGS  is  an  effective 
tool  for  improving  INTSUM  composition,  and  it  is  our  goal  to  have  the  AGS  installed  within 
CFSMI  so  that  it  can  be  used  by  students  as  part  of  their  training  program. 
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2  The  Automated  Grading  System  (AGS) 


2.1  Detailed  Description  and  Manual  of  the  AGS  for 
Administrators 


In  this  document  the  words  “instructor”  and  “admin  user”  are  used  interchangeably  to  describe 
the  admin  user.  The  Automated  Grading  System  uses  Semantic  Models  that  generate  estimates  of 
similarity  for  pairs  of  documents.  In  this  way,  the  AGS  can  compare  and  mark  student  summaries 
against  a  “Gold  Standard”  summary  written  by  an  instructor.  The  instructor  can  also  include 
several  other,  smaller  gold  standard  documents  (which  we  refer  to  as  nuggets)  that  represent 
particular  pieces  of  information  that  must  be  present  in  a  passable  summary.  The  semantic  models 
need  to  be  trained  on  a  document  set  or  coipus  that  contains  the  background  knowledge  for  a 
given  summary  topic.  This  background  knowledge  is  required  so  that  the  AGS  can  make  accurate 
comparisons  between  the  student  and  instructor  summaries  given  a  certain  topic  and  provide 
immediate  feedback  to  the  student  user.  As  an  administrator  user  of  the  AGS,  system 
requirements  are  minimal.  The  AGS  is  a  web  based  application  that  works  well  with  the  major 
web  browsers:  Internet  Explorer  (v.7),  Firefox  (2. 0.0. 6)  and  Opera  (9.10).  The  AGS  system 
requires  that  both  JavaScript  and  pop-ups  be  allowed  for  the  AGS  website  on  the  admin  user’s 
web  browser. 


2.1.1  Semantic  Models  Contained  in  the  AGS  System 


As  mentioned  in  the  previous  section,  the  AGS  creates  semantic  representations  for  the 
INTSUMs  generated  by  students,  and  compares  them  to  a  gold  standard  document.  The 
psychological  and  computational  science  literatures  have  produced  several  models  for  extracting 
semantic  content  from  unformatted  text  in  an  unsupervised  fashion.  Roughly  speaking,  regardless 
of  the  underlying  algorithm,  the  models  work  on  the  notion  that  words  which  are  semantically 
related  will  tend  to  occur  in  similar  contexts,  or  documents.  The  models  are  trained  on  large 
document  collections  (perhaps  in  the  10’s  of  thousands  of  documents)  and,  from  its  discovery  of 
what  words  do  occur  in  the  same  contexts/documents  in  the  training  coipus,  makes  educated 
guesses  about  words  should  occur  together. 

The  models  available  to  the  AGS  are  listed  below  along  with  an  article  reference  to  allow 
interested  readers  to  learn  more  about  the  models. 

1)  Vector  Space  Model  ( Vectorspace ,  [1]) 

2)  Latent  Semantic  Analysis  (LSA,  [2]) 

3)  Topics  model  ( Topics ,  [3]) 

4)  Sparse  Non-negative  matrix  factorization  (SpNMF,  [4]) 
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5)  Independent  Components  Analysis  ( ICA ,  [5]) 

6)  Sparse  ICA  {SICA,  [6]) 

7)  Syntagmatic  Paradigmatic  model  {SP,  [7]) 

8)  Constructed  Semantics  Model  ( CSM ,  Kwantes,  2005) 

The  AGS  uses  LSA  by  default  to  create  semantic  representations  for  the  gold  standard  documents 
and  student  summaries.  LSA  generates  vector  representations  for  words.  The  vectors 
corresponding  to  the  words  in  a  document  are  summed  to  create  a  vector  representation  for  a 
document.  Once  formed  into  vectors,  the  match  between  the  two  is  measured  by  calculating  the 
vector  cosine  between  the  vectors  for  each.  A  vector  cosine  is  much  like  a  correlation  coefficient 
in  that  a  cosine  of  1.0  indicates  that  two  vectors  have  identical  trajectories  in  space,  and  are 
therefore  very  highly  similar,  and  a  cosine  of  0  indicates  that  the  two  vectors  are  unrelated.  The 
vector  of  a  good  summary  therefore,  will  have  a  much  higher  cosine  with  the  gold  standard 
documents  than  a  poor  summary. 

2.1.2  Getting  Started  with  the  AGS  System 
2. 1.2.1  Logging  in 

As  an  admin  user  of  the  AGS  system  you  will  have  already  been  supplied  with  a  username  and 
password.  To  log  into  the  AGS  system  enter  your  email  address  and  password  into  the  login  box 
on  the  left-hand  side  of  the  AGS  login  screen  (see  Figure  1). 


AGS  -  Automated  Grading  System 


Login  to  AGS 


Email  address: 
Password: 


Not  registered?  Please  fill  in  your  details  below. 

First  name:  | 

Surname:  | 

Email  address:  | 

Choose  a  password: 

[  Register  ] 


Figure  1  AGS  login  page. 
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2.1 .2.2  Finding  Your  Way  Around 


As  an  admin  user  of  the  AGS  system  you  have  access  to  the  Assignments,  Corpora,  Documents, 
Admin,  and  Help  Sections  of  the  AGS.  These  sections  are  accessed  by  clicking  on  the  tabs  at  the 
top  of  the  user  interface  (Figure  2).  These  sections  will  be  described  below  in  more  detail.  To 
logout  of  the  AGS  at  any  time,  click  the  “logout”  link  located  underneath  the  AGS  logo  on  the  top 
right  of  the  page. 


Assignments 


Corpora 


Help 


AGS  -  Automated  Grading  System 

Logged  in:  administrator@ags.com  (logout) 


ASSIGNMENT  WIZARD  -  Create  a  new  assignment  on  the  AGS.  You  can  use  this  wizard  to  enter  HISTORY 

the  assignment  cetails.  Note  you  can  "save"  at  any  time  then  manually  edit  the  details  at  a  Mon.  25  Aug  2008  14  54  50  -  Login  to  AGS 


sicnment  .'.izarc 


Mon.  25  Aug  2008  14  54  13  -  Logged  out  of  AGS 
Mon,  25  Aug  2008  14  50  48  -  Login  to  AGS 
Mon,  25  Aug  2008  14  5018  -  Logged  out  of  AGS 

Mon.  2S  Aug  2008  14  16  34  •  Logout  and  then  back  in  to  AGS  access  ne 
Mon.  25  Aug  2008  14  16  34  -  administiator@ags.com  upgraded  to  Adm 
Mon.  25  Aug  2008  14  16:03  -  Registered  with  AGS 


ASSIGNMENTS 


[  Edit  Selectee  | 

Assignment  Title 

□  Games  and  Warfare 


Remove  Selecteo  ] 

Created  Active 

Sun.  24  Aug  2008  16:16  08  / 


Figure  2  AGS  administrator  user  interface  (Assignments  Page) 


2.1 .2.3  Anatomy  of  an  Assignment 

AGS  assignments  are  composed  of  four  major  components: 

1)  A  corpus  or  knowledge  base,  a  set  of  documents  which  act  as  background  knowledge  for  a 
particular  assignment  topic.  The  effectiveness  of  the  AGS’s  ability  to  automatically  grade  student 
assignment  is  completely  dependent  on  a  good  match  between  this  corpus  and  the  domain  of 
knowledge  needed  to  accurately  access  the  similarity  between  Student  Summaries  and  “Gold 
Standard”  summaries  provided  by  the  Instructor. 

2)  User  Documents  are  documents  that  are  supplied  by  the  instructor  to  the  Student  User  to  read 
and  summarize. 
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3)  Instructor  Documents  or  “Gold  Standard”  Summaries  are  the  material  that  student  user’s 
summaries  are  compared  against  -  they  are  the  “Gold  Standard”  on  which  the  AGS  will  judge  the 
student  user  summaries.  That  is,  the  AGS  will  compare  Instructor  documents  with  Student 
Summaries  and  grade  these  documents  similarity  with  a  mark  between  0  and  1  (0  representing 
NO  similarity,  1  a  perfect  match). 

4)  Student  Summaries  of  the  User  Documents  that  are  submitted  to  the  AGS  system  by  the 
Student  User. 


2. 1.2. 4  Assignment  Prerequisites 

Before  creating  an  assignment  using  the  “Assignment  Wizard”  (see  Assignments  section  below), 
the  instructor  needs  to  load  both  the  User  Documents  and  Instructor  Documents  onto  the  AGS  (in 
some  cases  these  will  be  the  same  document).  Also,  if  a  corpus  file  or  knowledge  base  has  not 
already  been  loaded  onto  the  AGS  system,  then  instructor  will  need  to  load  an  assignment 
specific  coipus  onto  the  AGS  and  create  a  Space  using  the  AGS  tools  described  below. 


2.1.3  Corpora 

A  coipus  is  a  set  of  documents  that  provide  the  background  or  knowledge  base  for  the  Semantic 
Models  on  a  given  topics.  For  example,  if  the  instructor  wanted  students  to  summarise  “User 
Documents”  that  focused  on  fishing,  then  a  good  coipus  would  contain  documents  about  the 
various  types  of  fishing,  tackle,  fish,  and  so  on.  The  AGS  assumes  that  the  coipus  will  be 
provided  as  an  ASCII  file  containing  a  set  of  documents  each  separated  by  a  blank  line. 


2. 1.3.1  Loading  Corpora 

Click  on  the  “Corpora”  tab  of  the  main  menu  to  see  Coipora  page  displayed  in  Figure  3.  In  the 
top-left  box  labeled  “ADD  CORPORA”  click  the  “Browse”  button  to  locate  on  your  computer  the 
corpus  that  you  wish  to  upload  to  the  AGS.  Choose  whether  you  would  like  to  make  the  corpus 
Public  or  Private  (other  admin  users  will  not  be  able  to  access  Private  Coipora)  and  add  a 
description  of  the  corpus  as  this  may  help  you  identify  the  coipus  at  a  later  date.  Finally,  click 
“Upload”  to  move  the  corpus  from  your  computer  onto  the  AGS  system.  This  may  take  a  long 
time  depending  on  the  speed  of  the  network  and  the  size  of  the  corpus.  After  your  corpus  has 
finished  uploading  to  the  AGS  system  it  will  be  displayed  in  the  bottom-left  box  labeled 
“Available  Corpora”  on  the  Coipora  page.  To  use  the  newly  added  coipus,  it  first  needs  to  be 
compiled  into  a  “Space”  (see  section  below)  Note:  The  AGS  assumes  that  the  corpus  will  be 
provided  as  an  ASCII  file  containing  a  set  of  documents  each  separated  by  a  blank  line. 
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AGS  -  Automated  Grading  System 


Assignments  |  Corpora  |  Documents  Admin  Help  Logged  in:  administrator@ags.com  (logout) 

HISTORY 

Mon.  25  Aug  2008  14  54  50  -  Login  to  AGS 
Mon,  25  Aug  2008  14  54  13  -  Logged  out  of  AGS 
Mon,  25  Aug  2008  14  50  48  -  Login  to  AGS 
Mon,  25  Aug  2008  14  50  18  -  Logged  out  of  AGS 

Mon,  25  Aug  2008  14  16  34  -  Logout  and  then  back  in  to  AGS  access  new  privileges 
Mon,  25  Aug  2008  14  16  34  -  administrator@ags.com  upgraded  to  Admin  user 
Mon.  25  Aug  2008  14  16  03  -  Registered  with  AGS 


Note: 

1)  Large  corpora  files  can  take  several  minutes  to  upload,  this  depends  on  the  net.vork  speed. 
DO  NOT  browse  away  from  this  page  until  the  corpora  is  fully  uploaded. 

2)  A  Public  Corpus  can  be  accessed  by  all  admin  users,  but  only  removed  by  the  admin  >vho 
uploaded  the  corpus  to  the  AGS.  A  Private  Corpus  can  only  be  accessed  by  the  admin  who  put 
it  on  the  AGS. 

3)  Each  document  in  the  corpora  should  be  seperate  by  a  new  line.  Please  read  the  SEMMOD 
manual  in  the  help  section  for  more  information  on  corpus  pre-processing. 


AVAILABLE  CORPORA 

[  More  Details  | 

Join  Corpora  |  [  Create  Space  |  |  Rerr 

Corpus  name 

Created 

0 

bibleCleaned.cor 

Mon.  25  Aug  2008  16:30:13 

X 

n 

cefault.ccr 

Sat.  02  Aug  2008  07  24:05  (GMT) 

✓ 

□ 

tasa  IMDB  noEntities. 

cor  Wed.  06  Aug  2008  02:53:19  (GMT) 

✓ 

□ 

tasa  IMDB  noEntities 

NN...  Mon,  18  Aug  2008  20:32:14 

✓ 

ADD  CORPORA 


|  Browse...  1  [  Upload  | 


Permissions: 
Corpus  description: 


(j)  Public  O  Prival 


Figure  3  AGS  Corpora 


2. 1.3. 2  Compiling  a  Corpus  into  a  “Knowledge  Space” 

Corpus  files  that  have  been  added  to  the  AGS  by  instructors  must  be  compiled  into  a  “Space” 
before  they  can  be  used  in  any  assignments.  To  compile  a  corpus  into  a  Space,  select  the  check¬ 
box  located  before  the  corpus  filename  displayed  in  the  “AVAILABLE  CORPORA”  list.  Next, 
click  the  “Create  Space”  button,  and  choose  the  semantic  model  and  define  the  parameters  that 
you  want  to  use  when  compiling  the  Space.  The  AGS  system  defaults  to  Latent  Semantic 
Analysis  (LSA),  however  you  may  choice  between  several  models  for  your  Space.  To  choose  a 
different  model,  use  the  drop-down  menu  labeled  “Change  model. . .”  (see  Figure  4).  Either, 
accept  the  default  parameters  supplied  for  your  chosen  model  or  enter  new  values  as  required. 

Note.  When  dimensionality  is  a  parameter  for  a  model,  the  specified  number  of  dimensions  or 
topics  needs  to  be  equal  to  or  less  than  the  number  of  documents  contained  in  a  given  corpus.  If 
this  requirement  is  not  met,  an  error  message  will  be  displayed  in  the  History  Section  (right  hand 
panel  of  the  display)  outlining  the  need  to  fulfdl  this  requirement. 
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it  on  the  AGS. 

3]  The  AGS  assumes  that  the  corpus  will  be  provided  as  an  ASCII  file  containing  a  set  of 
documents  each  separated  by  a  blank  line. 


Create  LSA  Space 

Corpus  Name: 
Extension 

Number  of  Dimensions 
Number  of  Iterations 
Random  Seed 


Figure  4  Choose  a  semantic  model  from  the  drop-down  menu. 


For  example,  if  LSA  is  chosen  by  the  Instructor,  a  box  entitled,  “Create  LSA  Space”  will  appear, 
in  this  box  enter  details  such  as  the  number  of  dimension,  number  of  iterations  and  random  seed 
to  use  when  compiling  the  space  (see  Figure  5).  If  you  are  unsure  what  to  enter,  use  the  default 
settings,  remembering  however  that  the  number  of  dimensions  needs  to  less  than  or  equal  to  the 
number  of  documents  contained  in  the  corpus.  After  filling  in  these  parameters  for  the  “Space”, 
click  “Submit”.  The  compilation  of  a  “Space”  can  take  a  long  time,  depending  on  the  size  of  the 
corpus.  For  example,  for  a  medium  size  corpus  (approximately  40,000  documents)  allow  up  to 
half  an  hour.  After  the  successful  compilation  of  the  “Space”  a  tick  will  appear  after  the  corpus 
filename  in  the  column  labelled  “Space”,  this  indicates  that  this  “Space”  can  be  used  in  an 
assignment. 


DO  NOT  browse  away  from  this  page  until  the  corpora  is  fully  uploaded. 

2)  A  Public  Corpus  can  be  accessed  by  all  admin  users,  but  only  removed  by  the  admin  who 
uploaded  the  corpus  to  the  AGS.  A  Private  Corpus  can  only  be  accessed  by  the  admin  who  put 
it  on  the  AGS. 

3)  The  AGS  assumes  that  the  corpus  will  be  provided  as  an  ASCII  file  containing  a  set  of 
documents  each  separated  by  a  blank  line. 


Create  LSA  Space 

Change  model...  v  | 

■p 

Corpus  Name: 

|  bibleCleaned.cor 

...compiling  space 

Extension 

.Isa 

Number  of  Dimensions 

300 

Number  of  Iterations 

1000 

Random  Seed 

4357 

L  Submit  J  [  Cancel  j 

bun.  14  L 
Sun.  14  C 
Sun.  14  C 
Sun,  14  C 
Sun.  14  C 
Sun.  14  C 
Sun.  14  C 
Sun.  14  C 
Sun.  14  C 
Sun.  14  C 
deleted  fi 
Sun.  14  C 
Sun.  14  C 
successfu 
Sun.  14  C 
Sun.  14  C 
successfu 
Sun.  14  C 


Figure  5  Enter  model  parameters  or  use  the  default  values  that  are  supplied. 
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2. 1.3. 3  Corpora  and  Space  Details 

Details  about  individual  Corpus  files  and  Spaces  can  be  displayed  (replacing  the  History  area)  to 
the  instructor  in  the  right  panel  of  the  Corpora  page  by  checking  the  tick  box  next  to  a  coipus’ 
name  and  clicking  on  the  “More  Details”  button  (see  Figure  6).  If  more  than  one  Space  has  been 
compiled  for  an  individual  corpus  file,  the  details  of  each  Space  can  be  accessed  using  the  drop¬ 
down  menu  supplied  in  “Space  Details”  section  (see  Figure  6).  If  no  Space  has  been  compiled, 
only  the  Corpus  file  details  will  be  displayed. 


Assignments 


Corpora 


Help 


AGS  -  Automated  Grading  System 

Logged  in:  admin_test@ags.com  floooutl 


ADD  CORPORA 


Permissions: 
Corpus  description: 


Upload 


(•)  Public  O  Private 


Note: 

1)  Large  corpora  files  can  take  several  minutes  to  upload,  this  depenfls  on  the  network  speed. 
DO  NOT  browse  away  from  this  page  until  the  corpora  is  fully  uploaded, 
i  2)  A  Public  Corpus  can  be  accessed  by  all  admin  users,  but  only  removed  by  the  admin  who 
uploaded  the  corpus  to  the  AGS.  A  Private  Corpus  can  only  be  accessed  by  the  admin  who  put 
it  on  the  AGS. 

3]  The  AGS  assumes  that  the  corpus  will  be  provided  as  an  ASCII  file  containing  a  set  of 
documents  each  separated  by  a  blank  line. 


CORPUS  DETAILS 

Corpus  name 

Permissions 
Description 

Created 

Size 

SPACE  DETAILS 

Creation  Time 
Model 

Number  of  Dimensions 
Number  of  Docs 
Number  of  Iterations 
Number  of  Terms 
Random  Seed 


bibleCleaned.cor 

admin_test@ags.com 

Public 

This  is  a  text  version  of  the  King  Jam 
Gutenberg  Project. 

Mon.  15  Dec  2008  19:44:23 
2.0  Mb 


s  Eible  retrieved  from  the 


AVAILABLE  CORPORA 


More  Details  ] 

|  Join  Corpora  |  (  Create  Space  | 

[  Remove  | 

Corpus  name 

Created 

Space 

0 

bibleCleanec.cor 

Mon.  15  Dec  2008  19:44:23 

✓ 

1  □ 

bic-corDus.cor 

Sat.  29  Nov  2008  13:40:58 

✓ 

1  — ^ 

■r 

Figure  6  Creating  a  new  assignment  with  the  Assignment  Wizard  9 


2. 1.3.4  Joining  Existing  Corpora 

Existing  corpora  may  be  concatenated  by  checking  the  tick  boxes  next  to  two  corpus  names 
clicking  on  the  “Join  Corpora”  button  in  the  “AVAILABLE  CORPORA”  section.  The  instructor 
will  then  be  prompted  to  supply  a  name  for  the  new  corpus.  After  the  corpora  are  joined,  the 
newly  created  file  will  be  listed  in  the  “AVAILABLE  CORPORA”  section. 

Note:  This  action  will  not  join  existing  Spaces  which  have  been  compiled,  only  the  corpora.  A 
Space  will  need  to  be  compiled  for  the  newly  joined  corpora. 

2. 1.3. 5  Removing  Corpora  and  Spaces 

BE  WARNED:  Checking  the  tick  box  next  to  a  corpus  name  in  the  “AVAILABLE  CORPORA” 
section  and  then  clicking  “Remove”  will  remove  both  the  corpus  file  AND  any  Spaces  that  have 
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been  compiled  using  this  coipus  file!  This  action  can  only  be  made  by  the  instructor  if  they  are 
the  owner  of  the  coipus  file.  Furthermore,  this  action  cannot  be  reversed  by  the  instructor. 


2.1.4  Documents 

There  are  two  types  of  document  that  the  Instructor  needs  to  load  onto  the  AGS  system:  User 
Documents  and  Instructor  Documents. 

User  Documents  are  documents  that  are  supplied  by  the  instructor  to  the  Student  User  to  read 
and  summarize.  Student  Users  accessing  your  assignment  will  be  presented  with  these  User 
Documents. 

Instructor  Documents  or  “Gold  Standard”  Summaries  are  the  material  that  student  user’s 
summaries  are  compared  against  -  they  are  the  “Gold  Standard”  on  which  the  AGS  will  judge  the 
student  user  summaries.  That  is,  the  AGS  will  compare  Instructor  documents  with  Student 
Summaries  and  grade  these  documents  similarity  with  a  mark  between  0  and  1  (0  representing 
NO  similarity,  1  a  perfect  match). 


2. 1.4.1  Loading  Documents 

Both  User  Documents  and  Instructor  Documents  are  loaded  to  the  AGS  in  the  “Documents” 
section  of  the  AGS.  Click  on  the  “Documents”  tab  of  the  main  menu  to  see  Document  page 
displayed  in  Figure  7.  In  the  top-left  box  labeled  “ADD  DOCUMENT”  click  the  “Browse”  button 
to  locate  the  document  that  you  wish  to  upload  to  the  AGS.  Choose  whether  you  would  like  to 
make  the  document  Public  or  Private  (Note:  User  Documents  must  be  made  Public)  and  add  a 
description  of  the  document  as  this  may  help  you  identify  the  document  at  a  later  date.  Finally, 
click  “Upload”  to  move  the  document  from  your  computer  onto  the  AGS  system.  After  your 
document  has  finished  uploading  to  the  AGS  system  it  will  be  displayed  in  the  bottom-left  box 
labeled  “Available  Documents”  on  the  Documents  page. 


Note:  All  documents  must  be  contained  in  an  ASCII  fde,  the  AGS  can  not  process  documents  in 
that  are  in  proprietary  formats  such  as  .doc  or  .pdf 
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Assignments 


Corpora 


Help 


AGS  -  Automated  Grading  System 

Logged  in:  administrator@ags.com  flocoutl 


ADD  DOCUMENT 

New  document  file: 

Permissions: 
Document  description: 


Browse. I  ]  [  Upload  ] 


®  Public  O  Private 


HISTORY 

Mon,  25  Aug  2008  14  54  50  -  Login  to  AGS 
Mon,  25  Aug  2008  14:54:13  -  Logged  out  of  AGS 
Mon,  25  Aug  2008  14  50  48  -  Login  to  AGS 
Men,  25  Aug  2008  14  50  18  -  Logged  out  of  AGS 
i  Mon.  25  Aug  2008  14: 16  34  -  Logout  and  then  back  i 
Mon.  25  Aug  2008  14  16  34  -  administrator@ags.con 
Mon.  25  Aug  2008  14  16:03  -  Registered  with  AGS 


i  to  AGS  access  new  privileges 
upgraded  to  Admin  user 


Note: 

1)  Large  document  files  can  take  several  minutes  to  upload,  this  depends  on  the  network 
speed.  DO  NOT  browse  away  from  this  page  until  the  document  is  fully  uploaded. 

2)  A  Public  Documents  can  be  accessed  by  all  admin  users,  but  only  removed  by  the  aomin  who 
uploaded  the  document  to  the  AGS,  A  Private  Documents  can  only  be  accessed  by  the  admin 
who  put  it  on  the  AGS. 

3)  Please  read  the  SEMMOD  manual  in  the  help  section  and  apply  the  information  on  corpus 
pre-processing  to  the  documents  uploaded  to  the  AGS. 


AVAILABLE  DOCUMENTS 


[  More  Details  |  [  Join  Documents  |  [  Remove  | 

Document  name  Created 

□  ancient  cames.txt  Mon,  18  Aug  2008  21: 14:49 

^  modern  aboricinal  artis...  Mon.  18  Aug  2008  20:25;  10 

□  >vw2  advances  in  technol...  Mon,  18  Aug  2008  20:25:04 


g 

u 


Figure  7  AGS  Documents 


2. 1.4. 2  Document  Details 

Details  such  as  ownership,  instructor  defined  description,  number  of  unique  words,  total  number 
of  words,  permissions  (Public  or  Private),  and  file  creation  date  can  be  displayed  to  the  instructor. 
This  information  is  accessed  by  checking  the  tick  next  the  relevant  document  and  clicking  “More 
Details”  in  the  “AVAILABLE  DOCUMENTS”  section.  This  document’s  details  will  then  be 
displayed  in  the  right  panel  of  the  screen,  replacing  the  History  data.  The  full  text  contained  in  a 
document  can  be  viewed  by  clicking  on  the  file  icon  aligned  with  the  file  of  interest;  this  is 
located  under  the  “View”  label  in  the  “AVAILABLE  DOCUMENTS”  section.  The  document 
will  then  be  displayed  in  the  right  panel  of  the  screen  (see  Figure  7). 


2. 1.4. 3  Joining  Documents 

Two  documents  can  be  concatenated  by  checking  the  tick  boxes  next  to  the  files  of  interest  and 
then  clicking  on  the  “Join  Documents”  button  in  the  “AVAILABLE  DOCUMENTS”  section.  The 
instructor  will  then  be  prompted  for  a  new  filename  for  the  newly  joined  documents.  After  the 
join  process  is  complete,  the  newly  joined  document  file  will  be  displayed  in  the  “AVAILABLE 
DOCUMENTS”  section. 


10 


DRDC  Toronto  TM  2012-0600 


2. 1.4. 4  Removing  Documents 


Documents  can  be  removed  from  the  AGS  system  by  checking  the  tick  box  next  to  their  filename 
in  the  “AVAILABLE  DOCUMENTS”  section,  and  then  clicking  on  the  “Remove”  button.  Only 
one  document  can  be  removed  at  a  time. 


AGS  -  Automated  G 


Assignments  Corpora  Documents  Admin  Help 


Logged  in  admin_ 


ADD  DOCUMENT 


New  document  file: 


1  Browse.. .  ]  [  Upload  | 


Permissions: 


®  Public  O  Private 


Document  description: 


Note: 

1)  Large  document  files  can  take  several  minutes  to  upload,  this  depends  on  the  network 
speed.  DO  NOT  browse  away  from  this  page  until  the  document  is  fully  uploaded. 

2)  A  Public  Documents  can  be  accessed  by  all  admin  users,  but  only  removed  by  the  admin  who 
uploaded  the  document  to  the  AGS.  A  Private  Documents  can  only  be  accessed  by  the  admin 
who  put  it  on  the  AGS. 

3)  Documents  must  be  provided  as  an  ASCII  file 


AVAILABLE  DOCUMENTS 

l  More  Details  ~]  [  Join  Documents  ]  [  Remove  ] 

Document  name  Created 

[^1  tech  aavances  ■•■•w2.txt  Tue.  26  Aug  2008  23:00:41 


g] 


DOCUMENT  DETAILS 

Document  name 

Description 

Created 

Unique  terms 

Word  count 

Owner 

Size 

Permissions 


tech_advance  s_ww2 .  txt 
An  essay  I  found  on  the  web 
Tue.  26  Aug  2008  23:00  41 
27  9 
499 

admin_test©  ags.com 

2.9  Kb 

Public 


Figure  8  AGS  Documents  Page  -  More  Details 


2.1.5  Assignments 


|  Assignments  |  Corpora  Documents  Admin  Help 

ASSIGNMENT  WIZARD  -  Create  a  new  assignment  on  the  AGS.  You  can  use  this  wizard  to  enter 
the  assignment  details.  Note  you  can  "save"  at  any  time  then  manually  edit  the  details  at  a 


AGS  -  Automated  Grading  System 

Logged  in  administrator@ags.com  floaoutl 


HISTORY 

Mon.  25  Aug  2008  14.54.50  -  Login  to  AGS 
Mon.  25  Aug  2008  14  54:13  -  Logged  out  of  AGS 
Mon.  25  Aug  2008  14.50:48  -  Login  to  AGS 
Mon,  25  Aug  2008  14  50  18  -  Logged  out  of  AGS 

Mon,  25  Aug  2008  14: 16  34  -  Logout  and  then  back  in  to  AGS  access  new  privileges 
Mon.  25  Aug  2008  14: 16  34  -  administrator@ags.com  upgraded  to  Admin  user 
Mon.  25  Aug  2008  14:16:03  •  Registered  with  AGS 


ASSIGNMENTS 

[  Edit  Selectee  | 

[  Remove  Selectee  ] 

Assignment  Title 

Created 

Active 

~|  Games  and  Warfare 

Sun.  24  Aug  2008  16:16:08 

✓ 

Figure  9  Creating  a  new  assignment  with  the  Assignment  Wizard 
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2. 1.5.1  Creating  a  New  Assignment 

After  the  necessary  assignment  documents  and  corpus  have  been  loaded  onto  the  AGS  (see 
Corpora  and  Documents  sections  above),  the  instructor  can  use  the  Assignment  Wizard  to  create  a 
new  assignment.  Click  on  the  “Assignments”  tab  of  the  top  main  menu  to  see  Assignments  page 
displayed  in  Figure  9.  In  the  top-left  box  labeled  “ASSIGNMENT  WIZARD”,  click  on  the  “Start 
Assignment  Wizard”  button  to  open  the  Assignment  Wizard. 

Note:  At  each  stage  of  the  assignment  creation  process  the  Assignment  Wizard  will  prompt  the 
instructor  for  information  about  the  assignment.  A  description  of  the  information  required  by  the 
AGS  and  addition  tips  are  also  displayed  on  the  screenshot  (circled  in  red  in  Figure  10). 


Figure  10  The  Assignment  Wizard 


2. 1.5.2  Assignment  Summary  Page 

After  the  instructor  has  completed  filling  in  the  assignment  information  that  the  Assignment 
Wizard  requested,  an  Assignment  Summary  is  displayed  (see  Figure  1 1).  From  this  screen  the 
instructor  can: 

1)  Activate  the  Assignment  -  only  Active  assignments  will  be  accessible  to  Student  Users  of  the 
AGS.  To  make  an  assignment  active  click  on  the  “Activate  Assignment”  button. 

2)  Edit  Assignment  components  -  select  an  assignment  component  by  clicking  the  radio  button 
located  before  its  label  (e.g.  Description),  then  click  on  the  “Edit  Selected”  button. 

3)  Access  Student  User  Results  -  click  on  the  “View  User  Results”  button. 

4)  Save  personal  notes  about  the  Assignment  -  enter  personal  notes  in  the  text  area  located  on  the 
right-hand  side  of  the  display,  then  click  on  the  “Save  Notes”  button. 
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5)  Delete  the  Assignment  -  click  on  the  “Delete  Assignment”  button. 

6)  Save  the  Assignment  -  click  on  the  “Save  Assignment”  button 


AGS  Assignment  Wizard  -  Assignment  Summary 


Title: 

Games  and  Warfare  ( X  Not  Active) 

@  Description: 

This  is  a  description  for  this  example  assignment.  This 
assignment  looks  at  the  correlation  between  ancient  games  and 
advances  in  technology  resulting  from  WW2.  The  description 
could  also  include  key  ares  for  student  users  to  focus  on  and 
expected  completion  dates. 

Semantic  Space: 

default.lsa 

o  u**r 

Documents: 

ancient_garr 

ies.txt 

es_in_technology.txt 

/~s  Summary  Word 
Limit: 

Min.  150 
Max.  1000 

Instructor 

Documents: 


ww2_advances_in_ 

-  artillery 


technology.txt 


Activate  Assignment 
|  Edit  Selected 


View  User  Results 


[  Delete  Assignment  | 
|  Save 


Instructor  Notes  I  Save  Notes  I 


Figure  11  Assignment  Wizard  -  Assignment  Summary > 


2. 1.5. 3  Student  User  Results 

To  pass  an  assignment  the  Student  User’s  summary  must  score  equal  to  or  greater  than  ALL  of 
the  “Pass  Cut  Off  Scores”  that  have  been  set  by  the  instructor.  As  mentioned  previously  in  this 
manual,  the  AGS  will  rate  the  similarity  of  the  Student  Summaries  to  the  Instructor  Documents 
(gold  standard  summaries)  on  a  scale  ranging  from  0  (no  similarity)  to  1  (high  similarity).  If  the 
Pass  Cut  Off  Score  for  a  particular  Instructor  Document  was  set  at  0.8  and  the  similarity  score 
calculated  by  the  AGS  between  that  document  and  the  Student  Summary  was  0.5,  then  the  user 
would  fail,  alternatively  a  similarity  score  of  0.8  or  greater  would  pass. 


Note:  A  Student  Summary  >  will  only  pass  if  it  matches  or  exceeds  the  Pass  CutoOff  Scores  set  by 
the  instructor  for  ALL  Instructor  Documents. 
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AGS  Assignment  Wizard  -  Assignment  Results:  Games  and  Warfare 

A 

B 

c 

D 

E 

F 

USER 

SUBMITTED  |  0.6 

0.8 

, 

° 

0.8 

■  0.9  ■ 

0.9  «  Cutoffs 

|  frank. citizen  ■  igmail.com 

Mon  25  Aug  2008  18:55:02  0.794 

0.991 

0.943 

0.987 

0.949 

0.910 

view  summary 

0.000 

0.000 

0.000 

0.000 

0.000 

view  summary 

newuser  ■  ags.com 

Mon  25  Aug  2008  18:56:12  0.792 

0.969 

0.946 

0.963 

0.954 

0.910 

view  summary 

Mon  25  Aug  2008  18:5  i:07  j  0.928 

0.665 

0.642 

0.639 

0.653 

0.635 

view  summary 

Mon  25  Aug  2008  18:51:19  0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

view  summary 

Figure  12  Instructor ’s  Assignment  Results  Page. 


As  can  be  seen  on  the  Assignment  Results  Page  displayed  in  Figure  12,  Student  Summaries  that 
did  not  reach  the  Instructor  Document  Pass  Cut  Off  Scores  failed  (indicated  by  red  text).  To  the 
Student  User,  a  fail  on  any  recorded  comparison  to  a  Instructor  Document  would  mean  an  overall 
fail  and  they  would  need  to  resubmit  their  Student  Summary.  The  Student  User  is  only  given  the 
overall  grade  PASS  or  FAIL.  To  view  a  Student  User’s  summary  in  a  new  window,  simply  click 
the  “view  summary”  link. 


2. 1.5. 4  Editing  an  existing  assignment 

Assignments  that  have  previously  been  added  to  an  instructor’s  profile  are  displayed  in  the 
“Available  Assignments”  section  of  the  Assignments  page  (see  Figure  13).  To  edit  an  assignment 
listed  in  this  section,  either  click  on  the  link  displaying  the  assignments  name  or  click  on  the 
check-box  located  before  the  assignments  name  and  then  click  on  the  “Edit  Selected”  button.  The 
instructor  will  then  be  presented  with  the  Assignment’s  Summary  page  (see  Figure  11). 


|  Assignments  |  Corpora  Documents 


Help 


ASSIGNMENT  WIZARD  -  Create  a 
the  assignment  details.  Note  you 


•v  assignment  on  the  AGS.  You  can  use  this  wizard  to  enter 
i  "save"  at  any  time  then  manually  edit  the  details  at  a 


Start  Assignment  Wizard 


AGS  -  Automated  Grading  System 

Logged  in:  administrator@ags.com  (logout) 

HISTORY 

Mon.  25  Aug  2008  19:20:33  -  Assignment  created:  Trends  in  Internet  Usage 

Mon,  25  Aug  2008  17 :51 :04  -  Assignment  created:  Games  and  Warfare 

Mon.  25  Aug  2008  17:39:10  -  Assignment  (Games  and  Warfare)  deleted  successfully 

Mon.  25  Aug  2008  16  43  56  -  Space  ("bibleCleaned.lsa")  for  "bibleCleaned.cor"  compiled 

successfully 

Mon.  25  Aug  2008  16  43: 10  -  Attempting  to  compile  LSA  space  ("bibleCleaned.lsa")... 

Mon,  25  Aug  2008  14  54  50  -  Login  to  AGS 
Mon.  25  Aug  2008  14  54:13  -  Logged  out  of  AGS 
Mon,  25  Aug  2008  14  50  48  -  Login  to  AGS 
Mon,  25  Aug  2008  14  50  18  -  Logged  out  of  AGS 

Mon.  25  Aug  2008  14: 16:34  -  Logout  and  then  back  in  to  AGS  access  new  privileges 
Mon,  25  Aug  2008  14  16:34  -  administrator@ags.com  upgraded  to  Admin  user 
Mon,  25  Aug  2008  14  16  03  -  Registered  with  AGS 


Figure  13  Editing  an  existing  assignment 
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2. 1.5.5  Removing  an  existing  assignment 

To  remove  an  assignment  from  the  AGS,  on  the  Assignments  page  in  the  “AVAILABLE 
ASSIGNMENTS”  section,  select  the  check-box  located  before  the  assignment,  and  then  click 
“Remove  Selected”.  This  will  completely  delete  the  Assignment  from  the  AGS,  which  will  not  be 
recoverable.  However,  both  the  documents,  corpus  files  and  Spaces  used  in  this  assignment  will 
still  be  available  to  the  instructor  on  the  AGS  system. 


2.1.6  Editing  Administrator  Details 

2.1 .6.1  Changing  your  administrator  user  password 

The  AGS  offer  the  admin  user  the  ability  to  change  their  current  password.  First  click  on  the 
“Admin”  menu  tab  at  the  top  of  AGS  interface  page.  In  the  top-left  box  of  the  Admin  page  (see 
Figure  14),  “admin  user  details”  such  as  first  name,  surname,  email  address  and  star-hidden 
password  are  display.  Next  to  the  star-hidden  password  is  a  button  labeled  “Change”.  Clicking  on 
the  “Change”  button  displays  a  form  that  request  the  admin  user  to  input  both  their  old  password 
and  a  new  password.  After  entering  these  details,  click  “Submit”  to  change  the  admin  user’s 
password. 


AGS  -  Automated  Grading  System 


Assignments  Corpora  Documents  |  Admin  |  Help 


Logged  in:  administrator@sgs.com  (logout) 


ADMIN  USER  DETAILS 

HISTORY 

Mon,  25  Aug  2008  19:20:33  -  Assignment  created:  Trends  in  Internet  Usage 

Mon,  25  Aug  2008  17:51:04  -  Assignment  created:  Games  and  Warfare 

First  name: 

Joe 

Mon,  25  Aug  2008  17:39:10  -  Assignment  (Games  and  Warfare)  deleted  successfully 

Mon,  25  Aug  2008  16:43:56  -  Space  ("bibleCleaned.lsa")  for  "bibleCleaned.cor"  compiled 

Surname: 

Bloggs 

successfully 

Enu.il, 

administrator@ags.com 

Mon,  25  Aug  2008  16  43  10  -  Attempting  to  compile  LSA  space  ("bibleCleaned.lsa")... 

Mon,  25  Aug  2008  14  54  50  -  Login  to  AGS 

Mon,  25  Aug  2008  14:54: 13  -  Logged  out  of  AGS 

Password: 

[  Change  ] 

Mon,  25  Aug  2008  14  50:48  -  Login  to  AGS 

Mon,  25  Aug  2008  14  50  18  -  Logged  out  of  AGS 

Last  login: 

Mon.  25  Aug  2008  14:54:50 

Mon.  25  Aug  2008  14  16  34  -  Logout  and  then  back  in  to  AGS  access  new  privileges 

Mon.  25  Aug  2008  14:16:34  -  administrator@ags.com  upgraded  to  Admin  user 

Mon.  25  Aug  2008  14: 16:03  -  Registered  with  AGS 

CREATE  NEW  ADMIN  USER 


First  name: 


Surname: 


p*“wo,d:  |  Submit  | 

Note: 

1)  Check  update  has  been  successful,  this  will  be  record  in  the  HISTORY  located  to  the  right. 

2)  If  the  new  admin  user  already  has  an  AGS  account  and  normal  user  privileges,  only  there 
priveleges  will  be  updated  to  admin  user.  That  is.  other  user  details  such  as  firstname. 
surname,  email,  and  password  will  not  be  changed  by  this  upgrade.  Therefore,  when  upgrading  a 
current  user  to  administrator  status,  it  is  only  necessary  to  enter  the  user's  email  address. 


Figure  14  Create  a  new  admin  user. 


2. 1.6. 2  Creating  a  new  administrator  user 

Admin  users  can  both  create  new  admin  users  and  upgrade  existing  student  users  to  admin  users. 
To  create  a  new  admin  user,  fill  in  the  new  admin  user’s  details  in  the  box  labeled  “Create  New 
Admin  User”  located  in  the  bottom-left  of  the  Admin  page  (see  Figure  14).  Then  click  submit, 
and  the  new  admin  user  will  be  able  to  login  to  the  AGS  with  these  details.  To  upgrade  an 
existing  student  user  to  the  level  of  admin  user,  it  is  not  necessary  to  enter  all  of  the  user’s  details 
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in  the  “Create  New  User”  box,  only  the  email  address  they  use  on  the  AGS.  Then  click  submit. 
The  next  time  the  student  user  logs  into  the  AGS  they  will  have  admin  user  privileges.  Note 
entering  any  other  details  for  the  student  user  in  this  box  will  not  change  these  details  on  their 
AGS  profile,  only  their  status  will  be  upgraded  to  admin  user  by  this  action. 
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3  Validation  Study  for  the  AGS 


3.1  Background 

The  purpose  of  this  study  was  to  measure  the  extent  to  which  the  AGS  improves  the  quality  of 
INTSUMs,  and  the  extent  to  which  the  AGS’s  assessment  of  the  INTSUM  quality  was  aligned 
with  a  human’s  assessment  of  the  documents.  For  the  experiment  reported  here,  human  grades 
were  provided  by  the  lead  instructor  at  CFSMI. 

3.2  Method 

3.2.1  Participants 

Twenty-one  undergraduate  students  at  Ohio  State  University  participated  for  course  credit.  Age 
and  demographic  information  was  not  collected. 

3.2.2  Materials 

3.2.2. 1  Training  Materials  for  the  AGS 

The  first  step  for  AGS  is  to  provide  it  with  a  sample  of  language  from  which  it  will  create 
semantic  representations  for  terms.  For  the  exercise  reported  here,  the  LSA  component  of  the 
AGS  was  trained  on  approximately  10,000  articles  scraped  from  the  Stratfor  geo-intelligence 
website.  The  AGS  was  also  provided  with  a  so-called  “join  list”  which  contained  the  names  of 
cities,  regions,  countries,  continents,  and  other  parts  of  the  world  with  multiple  terms  in  their 
names  (e.g.,  North  America).  The  purpose  of  the  join  list  is  to  ensure  that  the  LSA  component  of 
the  AGS  treats  North  America  as  a  single  object  with  a  semantic  representation  rather  than 
creating  a  semantic  representation  for  both  component  terms,  North  and  America. 

3. 2.2. 2  Materials  shown  to  participants 

Eight  newspaper  articles  on  the  topic  of  maritime  piracy  were  selected  from  the  open  source 
media  for  summarization  by  participants.  The  shortest  article  had  123  words,  and  the  longest  had 
867  words.  The  articles  used  in  the  experiment  are  not  reproduced  here  but  can  be  made  available 
upon  request. 

3. 2.2. 3  Materials  not  shown  to  participants 

A  Summary  article  was  written  by  the  lead  instructor  at  CFSMI  to  serve  as  the  gold  standard 
against  which  all  summaries  written  by  participants  would  be  compared.  The  gold  standard 
document  read  as  follows: 
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During  the  summary  period,  seven  vessels  were  reported  hijacked,  with  an  attempt  made  on  at 
least  two  others.  Two  vessels  were  reported  as  released.  The  Somali  pirates  currently  detain  at 
least  six  vessels  and  132  sailors. 

On  12  June  2009,  a  German-owned  cargo  ship  was  seized  off  the  coast  of  Oman,  the  first 
recorded  attack  in  its  territorial  waters.  On  Oct.  19  the  Chinese  bulk  carrier,  the  De  Xin  Hai, 
was  hijacked  north  of  the  Seychelles.  The  41,000  metric  ton  vessel  is  carrying  about  76,000  tons 
of  coal.  On  Monday  16  Nov  the  chemical  tanker  MV  Theresa  with  28  North  Koreans  on  board 
was  hijacked.  In  a  second  incident  the  same  day,  pirates  attacked  a  Ukrainian  cargo  ship. 
Private  security  guards  on  board  returned  fire,  wounding  two.  EU  says  the  Ukrainian  ship  was 
not  hijacked,  though  a  possible  pirate  spokesman  claims  that  it  was.  The  Greek-owned  300,000- 
ton  Maran  Centaurus  was  seized  on  29  Nov  2009  about  800  miles  off  the  coast  of  Somalia.  It  was 
carrying  a  cargo  of  crude  oil  and  28  crew  members.  The  British-owned  yacht,  the  Lynn  Rival, 
was  seized  in  the  Indian  Ocean  on  23  October.  Somali  pirates  are  demanding  a  4m  ransom  for 
the  release  of  the  crew,  British  couple  Paul  and  Rachel  Chandle. 

This  year,  Somali  pirates  hijacked  the  4800  ton  North  Korean  cargo  ship  MV  Rim  early  Feb.  3  in 
the  Gulf  of  Aden.  The  ship  was  traveling  outside  the  normal  transit  corridor  with  unknown  cargo 
and  was  not  communicating  with  maritime  authorities.  If  the  cargo  is  weapons  or  drugs,  which 
North  Korea  has  been  known  to  ship  to  other  countries,  it  may  be  seized  and  used  by  pirates  in 
lieu  of  ransom  demands.  On  4  March  2010,  according  to  AFP,  the  2,100-tonne  Spanish  vessel 
Albacan,  fishing  half  way  between  the  Seychelles  and  the  Kenyan  coast,  was  attacked  by  pirates 
in  the  Indian  Ocean.  It  was  approached  by  two  pirate  skiffs,  one  of  which  fired  a  rocket- 
propelled  grenade  that  exploded  on  the  deck  of  the  fishing  vessel.  "Private  security  armed 
guards  "fired  back  "over  the  heads  of  the  pirate  skiff  and  the  attackers  fled.  On  Mar  4  an  empty 
Jeddah-bound  Saudi  oil  tanker,  the  Al-Nisr  Al-Saudi  was  hijacked  resulting  in  a  $20  million 
ransom  demand  for  the  5136  ton  vessel  and  its  crew  of  13.  The  ship  is  believed  to  be  moored  near 
Garacad,  Somalia.  The  tanker  was  reportedly  outside  an  internationally  recognized  transit 
corridor.  In  an  undated  incident  reported  by  Arab  News,  a  Turkish  vessel  escaped  capture  due  to 
the  intervention  of  a  Saudi  patrolling  team. 

Somali  pirates  released  the  Italian-flagged  tugboat  Buccaneer  Aug.  10  2009.  Although  the  Italian 
government  denies  ransom  payment,  pirates  claimed  to  have  received  between  $4  and  $5  million. 
Pirates  also  freed  the  German  vessel  Hansa  Stavanger  and  its  crew  Aug.  3  2009  after  a  ransom 
of  $2. 7  million  was  paid.  Both  vessels  were  seized  in  early  April  2009 

In  addition  to  the  gold  standard,  the  experimenters  included  four  gold  “nuggets”.  Each  nugget 
represents  a  segment  of  text  containing  information  that  a  good  summary  must  have.  Each  nugget 
has  an  associated  “hint”  that  participants  read  to  find  out  what  kind  of  detail  might  be  missing 
from  their  summary.  The  need  for  having  nuggets  is  not  obvious,  but  they  are  important  because 
the  evaluation  of  the  quality  of  a  summary  is  based  on  the  semantic  similarity  between  it  and  the 
gold  standard  documents.  That  being  the  case,  it  is  possible  that  a  summary  can  miss  important 
details  and  still  have  high  semantic  similarity  to  the  gold  standard.  Measuring  the  summaries’ 
similarity  to  the  nuggets  allowed  the  instructor  to  determine  the  extent  to  which  important  pieces 
of  information  were  also  present  in  the  summary.  The  hints  and  nuggets  were  was  follows: 

Hint  1  :  What  was  the  nationality  of  the  pirates? 
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Nugget  1:  Somalia  officially  the  Somali  Republic  (Somali)  and  formerly  known  as  the  Somali 
Democratic  Republic  under  Socialist  rule,  is  a  country >  located  in  the  Horn  of  Africa.  Since  the 
outbreak  of  the  Somali  Civil  War  in  1991  there  has  been  no  central  government  control  over  most 
of  the  country's  territory.  Somalia  has  a  population  of  around  10  million  inhabitants.  About  85% 
are  ethnic  Somalis,  who  are  traditionally  nomadic  pastoralists  and  have  historically  inhabited  the 
northern  part  of  the  country > 

Note:  The  text  of  Nugget  1  was  copied  from  Somalia’s  entry  in  Wikipedia 
Hint  2:  What  are  the  nationalities  of  the  ships  that  were  hijacked? 

Nugget  2:  Europe  Asia  European  Asian  German  Germany  words  Chinese  China  North  Korea 
North  Korean  Ukranian  Ukraine  Greek  Greece  British  Britain  England  English  UK  United 
Kingdom  Spanish  Spain  Turkish  Turkey  Italian  Italy 

Hint  3:  Where  did  the  hijackings  occur? 

Nugget  3:  Oman,  officially  called  the  Sultanate  of  Oman  is  an  Arab  state  in  southwest  Asia  on  the 
southeast  coast  of  the  Arabian  Peninsula  Oman  Seychelles.  Indian  Ocean  Aden  is  a  seaport  city 
in  Yemen,  located  by  the  eastern  approach  to  the  Red  Sea  the  Gulf  of  Aden,  some  170  kilometres 
east  of  Bab-el-Mandeb.  Aden  was  the  capital  of  the  People's  Democratic  Republic  of  Yemen  until 
that  country's  unification  with  the  Yemen  Arab  Republic 

Note:  The  text  of  Nugget  3  was  copied  from  Oman’s  entry  in  Wikipedia 

Hint  4:  What  were  the  conditions  of  release? 

Ransom  is  the  practice  of  holding  a  prisoner  or  prisoners  or  item  to  extort  money  or  property  to 
secure  their  release  or  it  can  refer  to  the  sum  of  money  involved.  A  consideration  paid  or 
demanded  for  the  release  of  someone  or  something  from  captivity 

Note:  The  text  of  Nugget  4  was  copied  from  Ransom's,  entry  in  Wikipedia 

3. 2. 2. 4  Experimental  Apparatus 

The  experiment  was  run  on  personal  computers  running  a  web-browser,  and  connected  to  the 
internet.  The  AGS  was  installed  on  a  server  at  Ohio  State  University. 

3.2.3  Procedure 

3.2.3. 1  Data  Collection 

Participants  were  tested  in  groups,  but  worked  individually.  Each  participant  had  his  or  her  own 
computer  on  which  to  work,  and  was  given  a  user  account  in  the  AGS.  After  a  brief  instruction 
session  on  how  to  use  the  AGS,  they  logged  on,  and  did  the  task.  Each  participant  read  the  eight 
newspaper  articles,  and  wrote  a  summary  inside  a  text  window  contained  in  the  AGS.  After 
completing  the  summary,  the  participant  clicked  on  a  button  labelled,  “submit  summary”  and  was 


DRDC  Toronto  TM  2012-060 


19 


immediately  provided  feedback  on  the  quality  of  their  summary.  Feedback  came  in  the  form  of  a 
set  of  five  cosines,  one  for  the  summary’s  similarity  to  the  gold  standard  and  one  for  each  of  the 
hints.  Participants  were  asked  to  edit  their  summaries  as  much  as  and  as  many  times  as  required 
to  maximize  the  five  cosines,  and  to  stop  when  they  were  satisfied  that  they  could  not  be 
improved  further. 

3. 2. 3. 2  Grading  of  Summaries 

Summaries  were  graded  using  two  means.  They  were  graded  by  the  AGS  via  the  LSA  module 
and  by  the  lead  instructor  at  CFSMI.  For  each  participant,  two  summaries  were  graded:  the 
summary  written  on  the  participant’s  first  attempt  and  the  participant’s  final  submission  after  they 
had  used  the  feedback  to  improve  it.  The  AGS  provided  five  scores  for  each  of  the  two 
summaries  submitted  by  participants.  The  scores  were  taken  as  the  vector  cosine  between  the 
summary  and  the  gold  standard,  and  to  each  of  the  four  nuggets  we  included. 

The  summaries  were  then  sent  to  the  lead  instructor  at  CFSMI  for  grading.  The  instructor  was 
given  the  21  pairs  of  summaries  (2  summaries  from  each  participant)  in  a  single  file.  Which 
summary  was  written  first  or  last  was  randomized  for  each  of  the  21  pairs.  The  instructor  was 
asked  to  grade  each  summary  on  a  10-point  scale  where,  a  score  of  10  represented  a  perfect 
summary  and  a  score  of  1  represented  an  exceptionally  poor  quality  one. 

3.3  Results  and  Discussion 

We  will  organize  this  section  of  the  report  into  a  set  of  questions  that  we  can  ask  regarding  how 
well  the  AGS  improved  INTSUM  creation. 


3.3.1  Were  the  grades  given  by  the  AGS  and  the  instructor  generally  in 
agreement? 

There  was  a  modest  but  reliable  relationship  between  the  AGS  and  the  instructor  for  grades  given 
to  essays  written  on  the  participants’  first  attempt,  r  =  .42,  t(20)  =  2.03,_p  <  .05.  In  other  words, 
generally  speaking,  the  instructor  and  the  AGS  agreed  upon  what  were  good  and  poor  initial 
attempts  at  writing  an  INTSUM.  The  agreement  between  the  two  sources  of  grades  dropped  for 
the  final  summary  submission,  r  =  .17,  t(20)  =  0.75,  p  >  05.  The  drop  in  agreement  is  not 
surprising — a  strong  correlation  requires  a  healthy  amount  of  variability  in  the  grades  being 
compared.  As  all  the  INTSUMs  improved  with  practise,  the  variability  in  the  grades  decreased, 
thus  reducing  the  relationship  between  grade  sources. 

3.3.2  Did  the  AGS  and  the  instructor  agree  that  INTSUMs  got  better  with 
practice? 

Yes.  Virtually  all  participants’  INTSUMS  improved  with  practice.  Importantly,  the  improvement 
was  noted  by  both  the  AGS  and  the  instructor.  Vector  cosines  comparing  summaries  written 
before  training  to  the  gold  standard  [M  =  .94)  were  consistently  lower  than  the  cosines  for  those 
summaries  written  after  training  (M  =  .95),_t(20)  =  3.62,  p  <  .01.  As  well,  the  instructor’s  average 
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grade  out  of  10  for  summaries  written  before  practice  was  lower  (M  =  3.8)  than  that  for  those 
written  after  practice  and  feedback  (M  =  4.9),  t(20)  =  5.16,  p  <  .01. 

To  measure  the  degree  of  agreement,  we  calculated  a  Spearman’s  rank  correlation,  or  Rho, 
between  the  improved  grades  noted  by  both  for  every  participant.  The  Spearman’s  rank 
correlation  basically  measures  the  extent  to  which  big  the  AGS  and  instructor  agreed  on  which 
participants  experienced  large  improvements  in  their  INTSUM  quality  and  which  had  only  little. 
Across  the  21  participants,  the  Spearman  correlation  was  strong,  p  =  .58.  Hence,  the  AGS  and 
instructor  also  agreed  upon  the  amount  of  improvement  that  participants  had  as  a  result  of  training 
and  feedback. 

3.3.3  Did  the  amount  of  improvement  depend  on  the  amount  of  practice 
participants  had? 

No.  There  was  no  reliable  relationship  between  the  number  of  INTSUM  iterations  participants 
went  through  before  submitting  and  the  final  grade  they  received  from  either  the  instructor  or 
AGS.  In  other  words,  while  practice  had  a  significant  effect  on  the  quality  of  the  INTSUMs,  the 
amount  of  practice  seemed  to  matter  very  little. 
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4  Conclusions  and  Recommendations 


The  AGS  was  developed  jointly  by  DRDC  and  CFSMI  to  provide  the  school  with  an  easy  means 
by  which  students  could  improve  summary-writing  skills  without  the  need  for  in-class  instruction. 
It  is  a  simple-to-use,  browser-based  tool  that  allows  students  to  practice  their  writing  ability  on 
any  computer  connected  to  the  network  upon  which  the  tool  is  installed.  We  have  included  the 
user’s  manual  for  students  in  the  appendix  of  this  report. 

The  small  validation  study  reported  above  establishes  that  the  AGS  is  an  effective  tool  for 
improving  the  quality  of  INTSUMs — both  the  AGS  and  human  grader  agreed  that  the 
participants’  summaries  improved  after  feedback.  Because  the  AGS  appears  to  be  an  effective 
tool  for  improving  summary/report  quality,  we  recommend  that  it  be  considered  for  inclusion  in 
of  a  suite  of  educational  tools  for  Canadian  Forces  members  enrolled  in  programs  that  require 
effective  writing  skills  from  its  graduates. 
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Annex  A  AGS  Student  User  Manual 


Automated  Grading  System  (AGS)  -  Student  User  Manual 

The  Automated  Grading  System  uses  Semantic  Models  that  generate  estimates  of  similarity  for 
pairs  of  documents.  In  this  way,  the  AGS  can  compare  and  mark  student  summaries  against  a 
“Gold  Standard”  summary  written  by  an  instructor.  The  semantic  models  need  to  be  trained  on  a 
document  set  or  coipus  that  contains  the  background  knowledge  for  a  given  summary  topic.  This 
background  knowledge  is  required  so  that  the  AGS  can  make  accurate  comparisons  between  the 
student  and  instructor  summaries  given  a  certain  topic.  In  summary,  it  is  possible  to  use  the 
semantic  models  contained  in  the  AGS  system  to  grade  student  summaries  and  provide  immediate 
feedback  to  the  student  user. 

System  Requirements 

As  a  student  user  of  the  AGS,  system  requirements  are  minimal.  The  AGS  is  a  web  based 
application  works  well  with  the  major  PC  web  browsers:  Internet  Explorer  (v.6+),  Firefox 
(2. 0.0. 6+)  and  Opera  (9.10+).  For  best  display,  it  is  recommended  that  the  student  user  utilizes  the 
most  current  version  of  the  web  browser.  The  AGS  system  requires  that  both  JavaScript  and  pop- 
ups  be  allowed  for  the  AGS  website  on  the  student  user’s  web  browser. 

Getting  Started  with  the  AGS 

On  the  right-hand  side  of  the  login  page  (Figure  1)  there  is  a  registration  box  for  new  student 
users.  Simply,  fill  in  your  first  name,  surname,  email  address,  and  choose  a  password.  After 
registration  is  complete  you  will  be  logged  into  the  AGS.  To  log  back  in  on  subsequent  occasions 
use  the  login  box  on  the  left-hand  side  of  the  main  page  using  your  email  address  and  password. 


AGS  -  Automated  Grading  System 


Login  to  AGS 


Email  address: 
Password: 


|  Login  | 


Not  registered?  Please  fill  in  your  details  below. 

First  name:  | 


Surname:  | 
Email  address:  | 


Choose  a  password: 


Register  | 


Figure  15  AGS  login  page. 
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Finding  your  way  around 

As  a  student  user  of  the  AGS  system  you  have  access  to  the  Assignments,  User  Details,  and  Help 
Sections.  These  sections  are  accessed  by  clicking  on  the  tabs  at  the  top  of  the  user  interface 
(Figure  2).  These  sections  will  be  described  below  in  more  detail.  To  logout  of  the  AGS  at  any 
time,  click  the  “logout”  link  underneath  the  AGS  logo  on  the  top  right  of  the  page. 


AGS  -  Automated  Grading  System 


Assignments  User  Details  Help 

NEW  ASSIGNMENTS  -  Search  for  new  assignments  to  add  to  your  AGS  profile. 


Search  for  new  Asssig 


ASSIGNMENTS 


Edit  Selectea 


D  C 


Remove  Selected 


Figure  16  AGS  user  interface  (Assignments  Page 


New  Assignments 

To  search  for  new  assignments  to  add  to  you  AGS  profde,  click  “Search  for  new  Assignments” 
button  situated  in  the  top  left  box  of  the  Assignments  page  (see  Figure  2).  A  new  screen  will 
appear  (see  Figure  3),  that  contains  a  drop  down  menu  that  allows  the  student  user  to  browse 
through  the  active  assignments  that  have  been  set  by  instructors.  To  help  find  a  particular 
assignment,  instructor  names  have  been  included  with  assignment  names  in  this  drop  down  menu. 
After  selecting  an  assignment  from  the  drop  down  menu,  click  the  “Select  Assignment”  button  to 
access  the  assignment 
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AGS  -  Automated  Grading  System 


AGS  Assignment  Wizard 


Choose  assignment  from  list.. 

Choose  assignment  from  list.. 
jBens  assignment  (Instructor:  Ben  Stone) 

|tegs  (Instructor:  Tegwen  King) 

test  (Instructor:  Ben  Stone)  _ . 

i  ne  assignment  i  rae  win  allow  users  to  identity  and  select  this 
assignment.  These  titles  must  be  unique.  This  can  be  problematic  if 
another  administrator  has  already  used  the  title  you  wish  to  use, 
however  if  both  titles  were  allowed,  then  this  would  cause  confusion 
for  the  Users  when  completing  your  assignments.  If  you  choose  a 
Assignment  Title  that  is  already  taken,  the  AGS  will  inform  you  of  this 
and  request  another  title. 

TIPS 

To  avoid  Assignment  Title  conflicts,  one  naming  convention  could  be: 
[Assignment  Title]  =  [Admin  Surname]  +  [Useful  Assignment  Title] 


Select  Assignment 


Figure  1 7  Finding  a  new  Assignment 


AGS  Assignment  -  Document  Summary 


Title: 

Games  and  Warfare 

Instructor: 

administrator  administrator 
administrator@ags.com 

Description: 

This  is  a  description  for  this  example  assignment.  This 
assignment  looks  at  the  correlation  between  ancient  games  and 
advances  in  technology  resulting  from  WW2.  The  description 
could  also  include  key  ares  for  student  users  to  focus  on  and 
expected  completion  dates. 

Summary  Word 

Min.  150 

L.m.t: 

Max.  1000 

User  Documents: 

ancient_games.txt 

ww2_advances_in_technology.txt 

Results: 

AGS  -  Automated  Grading  System 


|  Submit  Summary 

|  Close  | 


Document  Summary  1  Save  Summary  |  |  Clear  I 


Figure  18  Assignment  -  Document  Summary  page 


Assignments 
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After  selecting  a  new  assignment  (as  described  in  the  previous  section)  the  student  user  will  be 
presented  with  the  “Assignment  -  Document  Summary”  page  (see  Figure  4).  This  is  the  work 
page  for  this  assignment.  It  contains  all  the  information  supplied  to  the  student  user  by  the 
instructor  who  created  this  assignment.  This  information  includes:  Assignment  Title,  Instructor 
Details,  Assignment  Description,  Instructor  selected  User  Documents  to  Summarize,  Student 
User  Summary  Word  length  constraints,  and  Results  of  previous  submissions  for  this  assignment. 


0  AC 

&  ' 


http://localhost:8080/AGS/scnpts/user nav menu.py 


Google  G  - 

v  Go  T*;^’  ft  Bookmarks  ▼  Pa9eRank  t  g  Popups  okay 

&  AGS 

~n 

AGS  Assignment  -  Document  Summary 


Games  and  Warfare 


rhis  is  a  description  for  this  example  assignment.  Tf 
issignment  looks  at  the  correlation  between  ancient 
rdvances  in  technology  resulting  from  WW2.  The  des 

expected  completion  dates. 


iS  ww2_advances_in_technology.txt  -  Windows  Internet  Explorer  ""UIbJB 


File  Edit  View  Favorites  Tools  Help 

Goos1e[[G> 


vIGoi-.^i  Q  »  ft  Bookmarks^  ®  Settings^ 


ww2  advances  in  technoloav.txt 


During  the  >var.  aircraft  continued  their  roles  of  reconnaissance,  fighters,  bombers  and  ground- 
support  from  World  War  I,  though  each  area  was  advanced  considerably.  Two  important 
additional  roles  for  aircraft  were  those  of  the  airlift,  the  capability  to  quickly  move  high-priority 
supplies,  equipment  and  personnel,  albeit  in  limited  quantities;[227]  and  of  strategic  bombing, 
the  targeted  use  bombs  against  civilian  areas  in  the  hopes  of  hampering  enemy  industry  and 
morale.[228j  Anti-aircraft  weaponry  also  continued  to  advance,  including  key  defences  such  as 
radar  and  greatly  improved  anti-aircraft  artillery,  such  as  the  German  88  mm  gun.  Jet  aircraft 
saw  their  first  limited  operational  use  during  World  War  II,  and  though  their  late  introduction  and 
limited  numbers  meant  that  they  had  no  real  impact  during  the  war  itself,  the  few  which  saw 
j  active  service  pioneered  a  mass-shift  to  their  usage  following  the  war, [229] 


At  sea,  while  advances  were  made  in  almost  all  aspects  of  naval 
j  of  development  were  focused  around  aircraft  carriers  and  submar 
the  war  aeronautical  warfare  had  relatively  little  success, [230]  a 
jthe  South  China  Sea  and  the  Coral  Sea  soon  established  the  can 
in  place  of  the  battleship.[231][232]  In  the  Atlantic,  escort  carri 

Mid- Atlantic  gap. [233]  Beyond  their  increased  effectiveness,  ca 
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raft[234]  and  their  not  requiring 


as  the  Leigh  light,  hedgehog,  squid,  and  homi 

Overland  warfare  changed  drastically  from  the  static  frontlines  experienced  during  World  War  I 
|  to  become  much  more  fluid  and  mobile.  An  important  change  was  the  concept  of  combined  arms 
warfare,  wherein  tight  coordination  was  sought  between  the  various  elements  of  military  forces; 
the  tank,  which  had  been  used  predominantly  for  infantry-support  in  the  first  World  War.  had 
jevolved  into  the  primary  weapon  of  these  forces  during  the  second. [238]  In  the  late  1930s,  tank 
:  design  was  considerably  more  advanced  in  all  areas  then  it  had  been  during  World  War  I. [239] 
and  advances  continued  throughout  the  war  in  increasing  speed,  armour  and  fire-power.  At  the 
start  of  the  war,  most  armies  considered  the  tank  to  be  the  best  weapon  against  itself,  and 
[developed  special  purpose  tanks  to  that  effect.[240]  This  line  of  thinking  was  all  but  negated  by 
jthe  poor  performance  of  the  relatively  light  early  tank  armaments  against  armour,  and  German 
[doctrine  of  avoiding  tank-to-tank  combat;  the  latter  factor,  along  with  Germany's  use  of 
I  combined  arms,  were  among  the  key  elements  of  their  highly  successful  blitzkrieg  tactics  across 
[Poland  and  France.[238]  Many  means  of  destroying  tanks,  including  indirect  artillery,  anti-tank  r 
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Figure  19  Instructor  supplied  User  Documents  for  students  to  summarize. 


User  Documents 

User  documents  are  supplied  by  the  instructor  for  students  to  summarize  for  each  assignment.  The 
student  user  can  access  these  documents  by  clicking  on  the  links  contained  in  the  “User 
Documents”  section  of  the  “Assignment  -  Document  Summary”  page.  By  clicking  these  User 
Document  links,  a  new  browser  window  is  opened  containing  the  corresponding  document  (see 
Figure  5)  .  NOTE:  Pop-up  windows  must  be  allowed  for  the  AGS  website  in  you  browser  settings 
to  enable  these  documents  to  be  display. 

Saving  a  Summary 

The  student  user  can  save  a  working  copy  of  the  current  summary  to  the  AGS  system  clipboard 
by  clicking  the  “Save  Summary”  button  located  above  the  text  area  on  the  right-hand  side  of  the 
“Assignment  -  Document  Summary”  page.  Saved  assignments  ARE  NOT  automatically 
submitted  for  grading.  Instead  they  are  kept  in  the  text  area  for  the  student  user  to  continue  using 
at  a  later  date.  Note:  A  Summary  first  be  saved  by  clicking  “Save  Summary:”,  before  it  can  be 
submitted  by  clicking  “Submit  Summary  ”. 

Submitting  a  Summary 
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The  Student’s  Summary  of  the  User  Documents  is  entered  into  the  “Document  Summary”  text 
area  on  the  right-hand  side  of  the  “Assignment  -Document  Summary”  page  (see  Figure  6).  After 
the  student  user  has  typed  the  summary  (or  copying  and  pasting  from  another  document)  into  this 
text  area  and  clicked  “Save  Summary”,  she  can  then  click  the  “Submit  Summary”  button  located 
in  the  center  of  the  page  to  have  it  graded  by  the  AGS  system.  After  grading  the  summary,  a 
PASS/FAIL  grade  will  then  be  display  next  to  a  link  to  a  copy  of  this  summary  in  the  “Results” 
section  on  the  left-hand  side  bottom  of  the  “Assignment  -  Document  Summary”  page.  Previously 
submitted  Student  Summaries  in  the  Results  section  can  be  viewed  by  clicking  on  the  links  in  this 
section.  Most  recently  submitted  Student  Summaries  will  be  displayed  at  the  top  of  the  Results 
section. 


AGS  Assignment  -  Document  Summary 


Title: 

new_assignment 

Instructor: 

admin  test 

admin_test@ags.com 

Description: 

The  Assignment  Description  is  displayed  to  users  when  selecting 
this  assignment.  It  is  the  administrator  message  to  the  users, 
containing  instructions  that  are  specific  to  this  assignment.  It 
may  contain  such  things  as  key  areas  of  focus  or  expected 
completion  dates.  Information  about  User  Summary  word  limits 
will  be  defined  by  the  administrator  later  in  this  assignment 
creation  process,  so  you  need  not  include  them  here. 

Summary  Word 

Min.  300 

Max.  1000 

User  Documents: 

tech_advances_ww2.txt 

Detailed  Results 


Close 


New  techniques  had  to  be  used  in  World  War  2  because  of  this  updated  technology.  It  was 
war  tactic  were  planes  were  first  sent  in  to  a  designated  area  and  bombed  it  quite  well, 
then  tanks  would  be  sent  through  to  destroy  things  that  withstood  the  bombing,  then 
finally  the  soldiers  were  sent  through,  hima  people  were  just  a  foreshadowing  of  the  ones 
to  come.  Mouse  holing  is  when  the  soldiers  would  blow  a  hole  in  the  wall  of  a  building  and 
move  through  the  building  capturing  the  nazi  soldiers  instead  of  going  out  on  the  open 
street  and  getting  picked  of  by  snipers.  Hopefully  people  within  our  World  can  learn  from 
the  mistake  of  this  type  of  warfare  and  leave  the  advancement  of  technology  to  helping 
people  instead  of  destroying  them.  These  planes,  which  were  quicker  and  more  advanced 
then  those  in  World  War  1,  carried  deadly  bombs  and  could  take  out  a  large  numbers  of 
people.  Also,  submarines  proved  deadly  as  they  were  out  of  radar  and  carried  deadly 
bombs  such  as  the  torpedo.  Techniques  such  as  Dmouse  holingd  and  lightning  warfares 
were  some  of  the  new  techniques  used. 


In  World  War  2  new  techniques,  weapons,  and  forms  of  transportation  were  introduced 
helping  this  war  to  have  even  more  causalities  then  the  First  World  War,  New  techniques 
had  to  be  used  in  World  War  2  because  of  this  updated  technology.  It  was  wartactic  were 
planes  were  first  sent  in  to  a  designated  area  and  bombed  it  quite  well,  then  tanks  would 
be  sent  through  to  destroy  things  that  withstood  the  bombing,  then  finally  the  soldiers 
were  sent  through,  hima  people  were  just  a  foreshadowing  of  the  ones  to  come.  Mouse 
holing  is  when  the  soldiers  would  blow  a  hole  in  the  wall  of  a  building  and  move  through  the 
building  capturing  the  nazi  soldiers  instead  of  going  out  on  the  open  street  and  getting 
picked  of  by  snipers.  Hopefully  people  within  our  World  can  learn  from  the  mistake  of  this 
type  of  warfare  and  leave  the  advancement  of  technology  to  helping  people  instead  of 
destroying  them.  These  planes,  which  were  quicker  and  more  advanced  then  those  in 


Figure  20  Submitting  a  summary  for  grading 


Clearing  a  Summary 

Clicking  on  the  “Clear”  button  located  above  the  text  area  will  remove  the  text  contained  in  the 
“Document  Summary”  text  area  from  the  page  and  from  the  AGS  system  clipboard. 

Closing  an  Assignment 

To  close  an  assignment,  simply  click  the  “Close”  button  located  in  the  center  of  the  “Assignment 
-  Document  Summary”  page. 


AGS  Assignment  -  Document  Summary 

Title: 

ww2  Tech 

Instructor: 

admin  test 

admin_test@ags.com 

< 

Description: 

Summary  Word 

Min.  150 

Limit: 

Max.  1000 

User  Documents: 

tech_advances_ww2.txt 

Results: 

ww2  Tech.000002.txt 

-  PASSED 

ww2  Tech.000001.txt 

-  PASSED 

Submit  Summary 


Detailed  Results 


Document  Summary  I _ sai 


Figure  21  Accessing  Result  Details 
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Detailed  Results 

The  Student  User  can  access  a  detailed  breakdown  of  how  their  summary  was  marked  against  the 
requirements  of  the  instructor.  To  do  this  click  the  “Detailed  Results”  button  located  in  the  middle 
of  the  “Assignment  -  Document  Summary”  page  (see  Figure  7).  The  Assignment  Results  page 
will  then  be  displayed  to  the  Student  User  (see  Figure  8).  On  the  Assignment  results  page,  a 
breakdown  is  given  on  how  each  summary  submitted  for  a  particular  Assignment  was  graded 
against  the  Instructor’s  criterion.  The  Student  User’s  summary  must  exceed  the  cutoff  scores  set 
by  the  Instructor  on  all  criteria  to  PASS.  All  scores  range  between  0  (lowest)  and  1  (highest).  An 
indication  of  the  topics  on  which  these  criteria  are  based  is  given  under  the  “HINTS”  section 
located  on  the  right-hand  side  of  the  Assignment  Results  page  (see  Figure  8).  Summaries  can  be 
view  by  clicking  the  “view  summary”  links  on  this  page. 


AGS  Assignment  Wizard  -  Assignment  Results:  new assignment 


A 


SUBMITTED 

HHHHSISHlSilSHI 

■HTfiteUM 

■BP  1*E  2001 


Figure  22  Assignment  Results 


Opening  an  existing  assignment 

Assignment  that  have  been  previously  been  added  to  a  student  user’s  profde  are  displayed  in  the 
“Active  Assignments”  section  of  the  “Assignments”  page  (see  Figure  9).  To  open  an  assignment 
listed  in  this  section,  either  click  on  the  link  displaying  the  assignments  name,  or  click  on  the 
check-box  located  before  the  assignments  name  and  then  click  on  the  “Open  Selected”  button. 


Assignments  j  User  Details  Help 

NEW  ASSIGNMENTS  -  Search  for  new  assignments  to  add  to  your  AGS  profile. 


[  Search  fi 


isignments 


AGS  -  Automated  Grading  System 

Logged  in:  newuser@ags.com  (logout) 

HISTORY 

Sun,  24  Aug  2008  16  49: 13  -  "Games  and  Warfare"  summary  submitted:  PASS 
Sun.  24  Aug  2008  16:21:30  -  "Games  and  Warfare"  assignment  added  to  profile 
Sun,  24  Aug  2008  16  05  01  -  "Bens  assignment"  assignment  added  to  profile 
Sun,  24  Aug  2008  16  04  33  -  Registered  with  AGS 


Figure  23  Opening  and  removing  existing  assignments 
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Removing  an  existing  assignment 

To  remove  an  assignment  from  the  Active  Assignments  listed  on  the  Assignments  page,  select  the 
check-box  located  before  the  assignment  and  then  click  “Remove  Selected”.  Removing  an 
assignment  only  deactivates  it  on  the  student  user’s  profile,  it  does  not  delete  the  assignment  or 
the  work  previously  completed  by  the  student  on  this  assignment  from  the  AGS.  To  re-activate  an 
assignment  that  has  been  removed,  select  it  again  from  the  assignment  list  which  is  displayed 
when  searching  for  a  new  assignment  (see  “New  Assignments”  section  above). 

User  Details  Section 

Changing  your  password 

The  AGS  offer  the  student  user  the  ability  to  change  their  current  password.  First  click  on  the 
User  Details  menu  tab  at  the  top  of  AGS  interface  page.  User  details  such  as  firstname,  surname, 
email  address  and  star-hidden  password  are  display.  Next  to  the  star-hidden  password  is  a  button 
labeled  “Change”.  Clicking  on  the  “Change”  button  displays  a  form  that  request  the  student  user 
to  input  both  their  old  password  and  a  new  password.  After  filling  in  these  details,  click  “Submit” 
to  change  the  student  user’s  password. 


Help  Section 

The  “Help  ”  section  of  the  student  users  AGS  interface  contains  a  copy  of  this  document.. 
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List  of  symbols/abbreviations/acronyms/initialisms 


DND 

Department  of  National  Defence 

DRDC 

Defence  Research  &  Development  Canada 

AGS 

Automated  Grading  System 

LSA 

Latent  Semantic  Analysis 

CFSMI 

Canadian  Forces  School  of  Military  Intelligence 

R&D 

Research  &  Development 

CDA 

Canadian  Defence  Academy 

IATF 

Influence  Activities  Task  Force 

RMCC 

Royal  Military  College  of  Canada 
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